Papers with decoder-only models
Annotation Entropy Predicts Per-Example Learning Dynamics in LoRA Fine-Tuning (2026.acl-srw)
Copied to clipboard
| Challenge: | Annotator disagreement on tasks like natural languageinference (NLI) reflects genuine linguistic ambiguity, but most fine-tuning recipes treat every example as equallylearnable. |
| Approach: | They ask whether annotator disagreement on tasks like natural languageinference (NLI) reflects genuine linguistic ambiguity. |
| Outcome: | The proposed method predicts learning dynamics on contested examples under LoRA. |
Garden Path Recovery in Causal and Masked Language Models (2026.acl-srw)
Copied to clipboard
| Challenge: | a linguistics study of garden-path sentences shows that recovery dynamics are important for linguistic evaluation . causal models show larger within-model disambiguation effects than masked models overall . |
| Approach: | They propose to compare garden-path recovery in causal and masked language models . they use 100 English garden- path/control pairs spanning three constructions . |
| Outcome: | The proposed model shows that decoder-only models exhibit sharper disruption at the point of syntactic revision, while encoders appear comparatively buffered at the disambiguator due to right-context access. |
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2025.acl-long)
Copied to clipboard
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, Iacopo Poli
| Challenge: | Encoder-only transformer models such as BERT offer a great performance-size tradeoff for retrieval and classification tasks compared to larger decoder models. |
| Approach: | They introduce a new transformer model, ModernBERT, which brings modern model optimizations to encoder-only transformer models. |
| Outcome: | The proposed model improves on the BERT transformer model and is faster and more memory efficient than the older models. |
Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on prompt engineering to extract sentence embeddings from large language models (LLMs) but these models are mostly decoder-only and the earlier tokens in the sentence cannot attend to the latter, resulting in biased encoding of sentence information and cascading effects on the final decoded token. |
| Approach: | They propose a plug-and-play and training-free technique that prepends each layer’s decoded sentence embedding to the beginning of the sentence in the next layer’ s input. |
| Outcome: | The proposed technique can significantly improve the performance of existing prompt-based sentence embedding methods across different LLMs while incurring negligible additional inference cost. |
Revealing Redundant Syntax in Large Language Models through Multi-Hop Dependency Paths (2026.findings-eacl)
Copied to clipboard
| Challenge: | Prior work on attention–syntax alignment has focused on single-hop Universal Dependency edges (DPs). |
| Approach: | They extract 2–3 hop MDPs from UD-parsed English and quantify head–relation alignment with an Unlabeled Attachment Score (UAS)-style metric modified for causal masking in decoder-only models. |
| Outcome: | The authors show that head alignments are overlapped and specialized . the head alignment is measurable in large language models trained on raw text . |
Interpretability-based Tailored Knowledge Editing in Transformers (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for modifying in-context learning fail to analyze the instability of in-constitu learning outcomes. |
| Approach: | They propose a model-based knowledge editing method that considers the unique information flow of each sample and aims to correct errors without costly retraining. |
| Outcome: | The proposed method exploits the critical role of feed-forward MLPs in decoder-only models and reveals diverse attribute recall across transformer layers, guiding edits to specific features at different depths and mitigating over-editing issues. |
LLaMAs Have Feelings Too: Unveiling Sentiment and Emotion Representations in LLaMA Models Through Probing (2025.acl-long)
Copied to clipboard
Dario Di Palma, Alessandro De Bellis, Giovanni Servedio, Vito Walter Anelli, Fedelucio Narducci, Tommaso Di Noia
| Challenge: | Large Language Models (LLMs) have become central to NLP, demonstrating their ability to adapt to various tasks through prompting techniques. |
| Approach: | They probe the hidden layers of Large Language Models to identify where sentiment features are most represented and to assess how this affects sentiment analysis. |
| Outcome: | The proposed approach enables sentiment tasks to be performed with memory requirements reduced by an average of 57%. |
Event Detection with a Context-Aware Encoder and LoRA for Improved Performance on Long-Tailed Classes (2026.findings-eacl)
Copied to clipboard
| Challenge: | Decoder-only models dominate the event detection literature, but their unidirectional attention mechanism has been a roadblock in getting strong performance on embedding. |
| Approach: | They propose to use Macro-F1 as a more representative measure of a model’s ability across the long-tail of event types to improve their models' performance. |
| Outcome: | The proposed model improves on the decoder-only models, showing that low-rank Adaptation can be an effective tool to enhance LLMs’ performance on long-tailed event classes. |
Sequence Repetition Enhances Token Embeddings and Improves Sequence Labeling with Decoder-only Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Modern language models (LMs) are trained in autoregressive manner, conditioned on the prefix. sequence labeling (SL) tasks assign labels to each individual input token, naturally benefiting from bidirectional context. |
| Approach: | They explore sequence repetition (SR) as a less invasive alternative to decoder-only models . they show that increasing the number of repetitions does not degrade SL performance . |
| Outcome: | The proposed technique improves the quality of token-level embeddings and surpasses encoders and unmasked decoders. |
Does Self-Rationalization Improve Robustness to Spurious Correlations? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Rationalization is fundamental to human reasoning and learning. |
| Approach: | They evaluate robustness to spurious correlations in encoder-decoder and decoder-only models . authors say explanations can come at the cost of robustness . |
| Outcome: | The proposed model outputs are more interpretable and easier to interact with for end-users than nonrationalizing models. |
Hit the Nail on the Head: Parameter-Efficient Multi-task Tuning via Human Language Intervention (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that PEFT on small pre-trained language models improves multitasking capabilities. |
| Approach: | They propose a multi-task learning framework that enables transfer of prior knowledge across tasks . they attach task descriptions to input samples and map them to task embeddings . |
| Outcome: | The proposed method improves performance on a T5 model and in decoder-only models . |
Bayelemabaga: Creating Resources for Bambara NLP (2025.naacl-long)
Copied to clipboard
| Challenge: | a lack of well-structured multilingual datasets remains a challenge for machine translation in under-resource languages. |
| Approach: | They propose to create a multilingual dataset for machine translation in the Bambara language, the vehicular language of Mali. |
| Outcome: | The proposed dataset is the most extensive curated multilingual dataset for machine translation in the Bambara language, the vehicular language of Mali. |
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022.emnlp-main)
Copied to clipboard
Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, Kristina Toutanova
| Challenge: | Pre-trained language models struggle on out-of-distribution compositional generalization . recent work shows considerable improvements on many NLP tasks from model scaling . |
| Approach: | They evaluate encoder-decoder models up to 11B parameters and decoder-only models up 540B parameters . they compare scaling curves for fine-tuning, prompt tuning, and in-context learning methods . |
| Outcome: | The proposed scaling methods improve compositional generalization on many tasks . fine-tuning generally has flat or negative scaling curves on out-of-distribution compositional . larger models are better at modeling the syntax of the output space, the study finds . |
Understanding Cross-Lingual Alignment—A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Cross-lingual alignment is the meaningful similarity of representations across languages in multilingual language models. |
| Approach: | They propose a taxonomy of methods to improve cross-lingual alignment . they argue that an effective trade-off between language-neutral and language-specific information is key . |
| Outcome: | The proposed methods can be applied to encoder models and encoder-decoder-only models . they show that language-neutral and language-specific information is key . |
Stereotype Detection as a Catalyst for Enhanced Bias Detection: A Multi-Task Learning Approach (2025.findings-acl)
Copied to clipboard
| Challenge: | a new study addresses bias and stereotypes in language models by exploring how learning them together improves performance. |
| Approach: | They propose a dataset for bias and stereotype detection that integrates religion, gender, socio-economic status, race, profession, and others. |
| Outcome: | The proposed dataset compares encoder-only models and fine-tuned decoder- only models . the results show that learning stereotypes together improves bias detection . |
Enhancing Goal-oriented Proactive Dialogue Systems via Consistency Reflection and Correction (2025.acl-long)
Copied to clipboard
| Challenge: | Unlike traditional dialogue systems, goal-oriented proactive dialogue systems focus on achieving specific objectives by actively guiding and anticipating user needs. |
| Approach: | They propose a model-agnostic two-stage Consistency Reflection and Correction framework that allows the model to reflect on discrepancies between generated responses and dialogue contexts and suggest possible corrections. |
| Outcome: | The proposed framework significantly improves the consistency between generated responses and dialogue contexts on three datasets. |
MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment (2025.findings-acl)
Copied to clipboard
Amir Hossein Kargaran, Ali Modarressi, Nafiseh Nikeghbal, Jana Diesner, François Yvon, Hinrich Schuetze
| Challenge: | Existing benchmarks for multilinguality for English-centric large language models focus on classic tasks or cover a minimal number of languages. |
| Approach: | They propose a method to assess multilingual capabilities of pre-trained LLMs using parallel sentences. |
| Outcome: | The proposed method evaluates the multilingual capabilities of pre-trained English-centric models using parallel sentences. |
Graph-Guided Textual Explanation Generation Framework (2025.emnlp-main)
Copied to clipboard
Shuzhou Yuan, Jingyi Sun, Ran Zhang, Michael Färber, Steffen Eger, Pepa Atanasova, Isabelle Augenstein
| Challenge: | Existing work has questioned their faithfulness, as they may not accurately reflect the model’s internal reasoning process regarding its predicted answer. |
| Approach: | They propose a Graph-Guided Textual Explanation Generation framework that generates a graph neural network layer that guides the NLE generation and generates explanations with greater semantic and lexical similarity to human-written ones. |
| Outcome: | The proposed framework improves NLE faithfulness by up to 12.12% compared to baseline methods on encoder-decoder and decoder-only models. |
ViLegalLM: Language Models for Vietnamese Legal Text (2026.findings-acl)
Copied to clipboard
| Challenge: | **ViLegalLM** is the first suite of Vietnamese pretrained language models for legal text understanding and generation. |
| Approach: | They propose a suite of Vietnamese pretrained language models for legal text understanding and generation. |
| Outcome: | The proposed models outperform instruction-tuned adaptation on four main Vietnamese legal downstream tasks. |
MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events (2026.acl-long)
Copied to clipboard
| Challenge: | Existing MLTC benchmarks are saturated and may be affected by training data contamination. |
| Approach: | They propose a machine learning benchmark based on medical device adverse event reports . they establish baselines across 20 encoder- and decoder-only models . |
| Outcome: | The proposed benchmarks show that small fine-tuned models achieve the strongest head-to-tail accuracy while maintaining competitive UQ. |